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(57)Abstract: 

PROBLEM TO BE SOLVED: To achieve a driving support by 
automatically displaying blind spot information, according to 
situations, with an image agreeable to a driver's view. 
SOLUTION: A driving support system is provided with an 
input unit 2 for various information including an image pickup 
device 21 , a control unit 1 for processing an image captured 
by the image pickup device, and a monitor 5 for displaying 
the processed image. The control unit is connected with a 
navigation unit 6 that can transmit road information. Based 
on the road information from the navigation unit and 
information from the input unit 2 for various information, the 
system makes a judgment on the state regarding 
approaching to an intersection without signal, driver's 
intention of iane change, and nighttime driving in the rain. 
Then, based on the judgment, an image in which the most appropriate road information is 
captured is automatically displayed on the monitor to support the driver case by case. 
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2.**** shows the word which can not be translated. 
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CLAIMS 

[Claim(s)| 

[Claim 1]Two or more imaging devices which are installed in vehicles and incorporate information 
on the external world. 

A control device which processes a picture which this imaging device captures. 
A monitor which displays a picture processed by this control device. 

Are the above the operation supporting device which it had, and said control device, It connects 
with a navigation device which can transmit a traffic information, approach in a crossing without a 
signal is judged based on a traffic information from this navigation device, and it has a picture 
selection means which displays an incorporation picture of information on a crossing road on a 
monitor automatically based on this judgment. 

[Claim 2]Have an input device which inputs a traveling condition and the contents of operation of 
vehicles into said control device as information, and said control device, The operation supporting 
device according to claim 1 which judges approach in a crossing which does not have said signal 
based on a traveling condition of vehicles inputted from an input device, and information on the 
contents of operation in addition to a traffic information from said navigation device. 
[Claim 3]The operation supporting device according to claim 2 which is the low vehicle speed 
information by a speed sensor with which information on a traveling condition of said vehicles is 
included by input device. 

[Claim 4]The operation supporting device according to claim 2 which is the brakes operation 
information by a brake sensor by which information on said contents of operation is included by 
input device. 

[Claim 5]The operation supporting device according to claim 2, 3, or 4 provided with an index 
displaying means which said control device generates and incorporates an index supplementary to 
a traffic information of an incorporation picture, and is superimposed in a picture. 
[Claim 6]The operation supporting device according to claim 5 in which said index is a graduation 
of rule-of-thumb distance on a ground surface from a self-vehicle. 

[Claim 7]The operation supporting device according to claim 6 with which said graduation is 
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generated by processing based on a traffic information from a navigation device according to the 
direction of a crossing road on an incorporation picture. 

[Claim 8]Two or more imaging devices which are installed in vehicles and incorporate information 
on the external world. 

A control device which processes a picture which this imaging device captures, and a monitor 
which displays a picture processed by this control device. 

Are the operation supporting device provided with the above, have an input device which inputs a 
traveling condition and the contents of operation of vehicles into said control device as information, 
and said control device, Connect with a navigation device which can transmit a traffic information, 
and based on a traffic information from this navigation device, the number of the slow lanes is 
plurality and it carries out on condition of formation of judgment of not approaching a crossing, 
either of the actions of vehicles obtained by traveling condition of vehicles inputted from an input 
device, information on the contents of operation, and image processing of a control device - empty 
vehicle line change, [judge and ] Based on judgment of this lane change, it has a picture selection 
means which displays an incorporation picture of information on vehicles back on a monitor 
automatically. 

[Claim 9]The operation supporting device according to claim 8 which is high vehicle speed 
information and operation information on a winker switch by a speed sensor with which a traveling 
condition of said vehicles and information on the contents of operation are included by input 
device. 

[Claim 10]The operation supporting device according to claim 8 with which an action of said 
vehicles is judged based on movement of an object recognized on a picture. 
[Claim 11]Two or more imaging devices which are installed in vehicles and incorporate information 
on the external world. 

A control device which processes a picture which this imaging device captures. 
A monitor which displays a picture processed by this control device. 

Are the operation supporting device provided with the above, have an input device which inputs a 
traveling condition and the contents of operation of vehicles into said control device as information, 
and said control device, Based on the contents of operation from an input device, visual 
recognition of a self-vehicle position on a road judges a difficult situation, and is provided with a 
picture selection means which displays an incorporation picture of information on the vehicles side 
on a monitor automatically based on this judgment. 

[Claim 12]The operation supporting device according to claim 10 which is the lighting information 
by operation of a lamplight switch in which said contents of operation are included by input device. 
[Claim 13]The operation supporting device according to claim 11 or 12 which is the wiper operating 
information by operation of a windshield wiper switch in which said contents of operation are 
included by input device. 

[Claim 14]The operation supporting device according to claim 1 1 , 12, or 13 with which said control 
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device includes an image processing means which makes an object on an incorporation picture 
conspicuous. 

[Claim 15]An operation supporting device of claim 11-14 given in any 1 paragraph which said 
control device judges an action of vehicles based on movement of an object recognized on a 
picture, and includes a warning means which superimposes warning information. 
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DETAILED DESCRIPTION 

[Detailed Description of the Invention] 
[0001] 

[Field of the InventionJThis invention incorporates the information on the vehicle circumference 
used as a driver's dead angle with two or more imaging devices, makes auto select of the required 
picture according to a traveling condition out of an incorporation picture, expresses it as a vehicle 
indoor monitor, and relates to the support device which complements a driver's field of view. 
[0002] 

[Description of the Prior ArtJSince a car has the body surrounding a driver's neighborhood, a 
driver's field of view produces the dead angle portion interrupted by Body Manufacturing Division 
except a window part. The indirect dead angle portion which has a field of view interrupted by them 
is also produced from the physical relationship of the surrounding geographical feature, a building, 
etc. also in the field of view which is not interrupted by the body. Dissolution of such a dead angle 
is a very important technical problem on reservation of safety. Then, a pure optical means is used 
from the former and various dead angle solving means using it are also further proposed with 
highly-efficient-izing and a miniaturization of the latest visual equipment so that the information on 
such a dead angle portion may be acquired indirectly. The art of an indication is one of those which 
are considered to be effective when acquiring many information which covers especially the dead 
angle range of a vehicle circumference in such a proposal at JP,5-310078,A- This conventional 
technology compounds the 1st and 2nd cameras that photo both sides of vehicles back, the 3rd 
and 4th cameras that photo both sides of a vehicle front, and the picture information obtained from 
them, and displays it on each of the display screen divided into four fields as a picture. 
[0003] 

[Problem(s) to be Solved by the lnvention]by the way, it cannot necessarily be said to be an 
effective method like the art of the above-mentioned proposal as the transfer technique of the 
information to the driver under a short-sleeved shirt situation to provide a driver with many 
information in parallel, but although required information can be narrowed down now in this art 
according to choosing one from the 1st thru/or the 4th picture information, and making it display on 
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the whole display screen, the driver himself has to perform that narrowing down and the thinking 
for it is required of a driver. 

[0004] It not being necessarily useful for support of operation that there is much the information, 
and requiring thinking of selection of information generally, when it provides a driver with dead 
angle information, A possibility of becoming rather troublesome, and useful information sometimes 
being overlooked, or it being avoided from the complicatedness of operation, and no longer being 
used becomes high. When the information is information which requires urgency, it is also harsh to 
force the selection upon a driver. Therefore, the information for driving support is selected carefully 
by what is suitable and indispensable according to a situation on that occasion, and, in the 
direction moreover obtained without exceptional operation, usefulness becomes high far. 
[0005]The inside of the information pictures of the imaging device which incorporates the vehicle 
circumference which this invention is made in view of such a situation, and serves as a driver's 
dead angle, It sets it as the main purpose to provide the operation supporting device which can 
select carefully the picture which can acquire the suitable information which agreed at the place 
which a driver means according to a situation, and can carry out a value monitor automatically. 
[0006] 

[Means for Solving the Problem]Two or more imaging devices which are installed in vehicles and 
incorporate information on the external world in order to attain the above-mentioned purpose. 
A control device which processes a picture which this imaging device captures. 
A monitor which displays a picture processed by this control device. 

Are the above the operation supporting device which it had, and said control device, It connects 
with a navigation device which can transmit a traffic information, approach in a crossing without a 
signal is judged based on a traffic information from this navigation device, and it has a picture 
selection means which displays an incorporation picture of information on a crossing road on a 
monitor automatically based on this judgment. 

[0007]ln order to raise judgment accuracy for an automatic display, Have an input device which 
inputs a traveling condition and the contents of operation of vehicles into said control device as 
information, and said control device, It is effective to have composition which judges approach in a 
crossing which does not have said signal based on a traveling condition of vehicles inputted from 
an input device and information on the contents of operation in addition to a traffic information from 
said navigation device. 

[0008]And as for information on a traveling condition of said vehicles, specifically, it is effective to 
consider it as low vehicle speed information by a speed sensor included by input device. 
[0009]As for information on said contents of operation, specifically, it is effective to consider it as 
brakes operation information by a brake sensor included by input device. 
[0010]As for said control device, in order to raise the usefulness of picture information displayed, it 
is also effective to have composition provided with an index displaying means which generates and 
incorporates an index supplementary to a traffic information of an incorporation picture, and is 
superimposed in a picture. 

http://ww4.ipdl.inpit.go.jp/cg^ 3/21/2008 



JP,2000-238594 5 A [DETAILED DESCRIPTION] Page 3 of 22 

[001 1]As for said index, specifically, it is effective to consider it as a graduation of rule-of-thumb 
distance on a ground surface from a self-vehicle. 

[0012]As for said graduation, it is effective to be generated by processing based on a traffic 
information from a navigation device according to the direction of a crossing road on an 
incorporation picture. 

[0013]Next, in an operation supporting device provided with two or more imaging devices which 
this invention is installed in vehicles and incorporate information on the external world, a control 
device which processes a picture which this imaging device captures, and a monitor which displays 
a picture processed by this control device, Have an input device which inputs a traveling condition 
and the contents of operation of vehicles into said control device as information, and said control 
device, Connect with a navigation device which can transmit a traffic information, and based on a 
traffic information from this navigation device, the number of the slow lanes is plurality and it 
carries out on condition of formation of judgment of not approaching a crossing, either of the 
actions of vehicles obtained by traveling condition of vehicles inputted from an input device, 
information on the contents of operation, and image processing of a control device - empty vehicle 
line change, [ judge and ] Based on judgment of this lane change, it has a picture selection means 
which displays an incorporation picture of information on vehicles back on a monitor automatically. 
[0014]ln this case, as for a traveling condition of said vehicles, and information on the contents of 
operation, specifically, it is effective to carry out operation information of high vehicle speed 
information by a speed sensor included by input device and a winker switch. 
[0015]As for an action of said vehicles, it is effective to be judged based on movement of an object 
recognized on a picture. 

[0016]Next, two or more imaging devices which are installed in vehicles and incorporate 
information on the externa! world. 

A control device which processes a picture which this imaging device captures. 
A monitor which displays a picture processed by this control device. 

Are the operation supporting device provided with the above, have an input device which inputs a 
traveling condition and the contents of operation of vehicles into said control device as information, 
and said control device, Based on the contents of operation from an input device, visual 
recognition of a self-vehicle position on a road judges a difficult situation, and is provided with a 
picture selection means which displays an incorporation picture of information on the vehicles side 
on a monitor automatically based on this judgment. 

[0017]ln this case, as for said contents of operation, it is effective to consider it as lighting 

information by operation of a lamplight switch included by input device. 

[0018]As for said contents of operation, it is also effective to consider it as wiper operating 

information by operation of a windshield wiper switch included by input device. 

[0019]As for said control device, it is also effective to include an image processing means which 

makes an object on an incorporation picture conspicuous. 

[0020]As for said control device, it is also effective to include a warning means which judges an 
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action of vehicles based on movement of an object recognized on a picture, and superimposes 

warning information. 

[0021] 

[Function and Effect of the inventionjln a crossing without a signal [ need / to be checked / the 
safety by grasp of the traffic situation of the road which crosses as compared with a crossing with a 
signal / in composition given in above-mentioned claim 1 / stili ], Effective driving support can be 
performed without giving a driver a burden and troublesomeness, since the dead angle information 
on the crossing road that a driver craves acquisition most can be displayed with automatically and 
sufficient timing. And since the existence of the signal has abolished the necessity of registering 
the database of crossing information peculiar to an operation supporting device, as information for 
an automatic display using the information on the navigation device registered beforehand, the 
composition of a device can be simplified. 

[0022]Since the traveling condition of vehicles and the information on operation which a driver's 
intention reflects in the judgment for an automatic display are added in the composition according 
to claim 2, Necessity judgment of the display which narrowed down the situation which a driver 
needs truly comes to be made, and a troublesome information noise can be lost for a driver. 
[0023]And in the composition according to claim 3, since the going-slowly state as reflection of 
approach in the badness and the crossing of a prospect is used as the concrete judgement factors 
of the traveling condition of vehicles, necessity judgment of a display is narrowed down further 
and, moreover, a display stage is also fitness-ized. 

[0024] In the composition according to claim 4, since brakes operation as reflection of approach in 
the badness and the crossing of a prospect is used as the concrete judgement factors of operation, 
necessity judgment of a display is narrowed down further and, moreover, a display stage is also 
further fitness-ized along with a driver's desire. 

[0025]ln the composition according to claim 5, it can supplement with the picture information which 
cannot deny that it is hard to judge the situation compared with direct viewing with the index which 
is easy to grasp intuitively, and a judgment of the road state from a monitor image can be made 
easy. 

[0026]ln the composition according to claim 6, it can supplement with a difficult sense of distance 
of grasp especially by distortion of a monitor image, and a road state judgment from a picture can 
be made still easier. 

[0027]An index display can be made to generate appropriately in the composition according to 
claim 7 according to a road state. 

[0028]Next, in the composition according to claim 8, effective driving support can be performed in 
the lane change under road run of two or more lanes, without giving a driver a burden and 
troublesomeness, since the dead angle information on traffic of the self-vehicles back that a driver 
craves acquisition most can be displayed with automatically and sufficient timing. And judgment of 
the number of the slow lanes being plurality and not approaching a crossing as information for an 
automatic display, Since the position of a lane number and a crossing has abolished the necessity 
of registering the database of a traffic information peculiar to an operation supporting device, using 
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the information on the navigation device registered beforehand, the composition of a device can be 
simplified. 

[0029] In the composition according to claim 9, since the high vehicle speed information and the 
operation information on a winker switch by the speed sensor as reflection of the intention of a lane 
change are used as concrete judgement factors, necessity judgment of a display is narrowed down 
further and, moreover, a display stage is also rationalized. 

[0030]ln the composition according to claim 10, even when a driver does not operate a winker 
switch, information required for a lane change can be provided at a proper stage. 
[0031]According to next, low visibility's etc. traffic condition and weather condition at the time of 
dazzling according to reflection of the headlight of the oncoming car at the time of a rainfall by the 
composition according to claim 1 1 at night, generating of the fog in daytime, and a heavy rain. 
Even when it is difficult to judge the check of the position of the self-vehicle to a travel iane or the 
road shoulder using the white line which shows the rain division and the center line on a road, It is 
not influenced by reflective because direction of an optic axis differs to a driver's look, Or the white 
line of self-vehicle nearness can be copied on a monitor using the incorporation image from the 
imaging device which cannot be easily influenced by a weather condition due to a thing near in 
position, and a driver can be provided with the picture information for recognizing the self-vehicle 
position over a travel lane top or the center line. 

[0032]And in the composition according to claim 12 or 13, a situation with difficult judgment of the 
self-vehicle position by the direct-vision private seal of a white line can be simply judged with the 
existing input device on vehicles. 

[0033]Even if a picture top can be recognized, it is on a monitor for a driver and can make it 
possible to recognize objects, such as a difficult white line of visual recognition, easily in the 
composition according to claim 14. 

[0034]ln the composition according to claim 15, the check of a self-vehicle position can make a 

more positive thing driving support under difficult low visibility's situation. 

[0035] 

[Embodiment of the invention]Hereafter, the embodiment of this invention is described over a 
drawing. The outline of the operation supporting device which applied the thought of this invention 
is shown in drawing 2 and draw ing 3. As a block shows, a system configuration to drawing 2 this 
device, The input device 2 which incorporates a variety of information required for control by 
making into a subject the control device 1 which contains the program which processes a picture, It 
comprises the selecting switch 3 which chooses suitably the information which a driver needs for 
operation, and further, it is connected to Navi ECU6 possessing the database 4 and the display 5 
of a navigation device so that information exchange with a navigation device may be possible. This 
uses the display 5 for the monitor for the information display of this device in this gestalt. About the 
selecting switch 3 in a figure, although a thing for exclusive use may be provided in this device, it is 
good also as a voice input method using the touch-panel method and voice recognition equipment 
using the display 5 of the navigation device. About a monitor, a thing for exclusive use may be 
provided separately and it can also be considered as a head up display method in that case. 
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[0036]CCD camera 21 (when writing it as a camera and distinguishing each camera hereafter) 
shown in a figure by 0 seal as an imaging device which constitutes one of the input devices as 
arrangement is shown in drawingjcable address C p L wjth g subscr|pt which rep|aces with numera|s and 

C rn C n( C„ „ C,. . . C. . _ - giving. The obstacle detecting sensor 28 
expresses a position, F R, R L, R R, M L, and M R 

(similarly) as a distance sensing device which it is installed in the center of four comers of vehicles, 
and both sides, and is shown by ** seal cable address S p L S p R S R L S R R S M L S M R , and S p M 

. S„ - qivinq - it is installed near [ the / same ] the position as the camera 21 , and also is 

and R M 3 

installed also in the center of a front-and-back-ends part. These distance sensing device 28 can 
also make it to consider it as the known device which carries out direct detection of the distance of 
a supersonic sensor, laser, a millimeter wave radar, etc., etc. the calculating means which finds 
distance indirectly by carrying out image processing of the incorporation picture of two or more 
above-mentioned cameras 21 within the control device 1 . 

[0037]!t is important requirements to provide so that a driver may coincide required picture 
information with the fieid-of-view feeling in the state where it is sitting on the driver's seat, when 
performing various operations on operation as a premise which performs driving support 
concerning the theme of this invention, therefore, the way the camera 21 should cling about an 
imaging device in order to realize it, in more detail, an installation attitude and an installed position 
are devised and the device is added to the method of presentation and display timing of the picture 
about the control device. These points are first explained one by one from the installation attitude 
of the camera 21. 

[0038](lnstallation attitude of an imaging device) As shown in drawing 1 , the dead angle range 
which does not look physical turns into a range which gave the slash to the figure surrounded by 
point D , D- ot D„ _ , and D . from the driver who sat on the driver's seat. However, the portion 

F L F R R R KL 

which is not visible by the pillar supporting the window frame or roof of a door is omitting the 
graphic display. On the other hand, in order to coincide with a driver's feeling the picture 
information which covers this range that is not in sight and is moreover obtained, as the 1st device 
of an installation attitude each camera 21, It is based on the posture which turned the upper part 
on the incorporation picture ahead of vehicles, and turned the optic axis to the vertical lower part, 
According to front and rear, right and left of an arranging position to vehicles, an inclination is 
attached in the direction of front and rear, right and left to the posture of a standard, and it installs 
with the posture in which copy the end of order about the edge of vehicles, i.e., order, and a left 
right side end part is copied about right and left at least. 

[0039]ln accordance with this meaning, about each camera 21 installed in a vehicles corner, it 
installs so that it may become in the comer of the self-body as an edge of the vehicles 
corresponding to that installed position, a vehicle circumference including that neighborhood, and 
the direction which looks down at the method of infinite distance simultaneously. In [ corner / of 
these vehicles ] especially the latest passenger car, Since the radius of circle is given from the 
device on a design, it is not necessarily ascertained, but with the corner of the vehicles told to this 
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invention. Mean the neighborhood including, the edge, i.e., outermost edge, when self-vehicles are 

seen at a flat surface, and more specifically, When using as a bumper the range which becomes 

the shape which can guess the direction in which the front end and the side edge or the back end, 

and the side edge of vehicles extend, for example, a corner, because the radius of circle of the 

angle of a bumper becomes small to the curvature almost near a straight line, the front end and the 

side edge or the back end, and the side edge of a direction to vehicles to which the front tip of a 

bumper and a side edge or a trailing edge, and a side edge extend - which it does not pass - 

the range which can guess whether it becomes a thing position is said. Therefore, as example of 

picture P , t , , u D P D1 A P D B are shown in drawing 4, The posture of 
K F L of the camera of each corner, F R, RL, and R R ~ s — 

each camera C_ . C c _ C D . . C D _ is set up contain in a picture to the range which becomes 

F L, F R, KL, and K K 

the shape which can guess extension of the front and back ends of the body, and right and left 
ends respectively. In a figure, as for cable address W L and W L R , the white line of a road surface, 

W*. , , and W. _ show the bumper as a self-vehicle corner on an image, as for the white line on an 

L L L R 

image, and numerals 90'. In this case, although the fitting location of the camera 21 is explained in 
full detail behind, generally, in each necessary part of the body, it installs in the highest possible 
place and a large field of view is secured. 

[0040]As the 2nd device of an installation attitude, horizontal direction of the camera 21 , The 
direction of a look as shown in drawin g 1 , in case a driver sits on a driver's seat and looks at the 
required direction, and a direction which carries out outline coincidence, That is, the point that it is 
installed in the posture in which the optic axis X turns to the direction of the vertical plane 
containing the straight line which connects the camera 21 to a driver, and the same direction as a 
parenchyma top, and the optic axis X crosses a ground surface at a relation with the posture by 
said 1st device becomes point A p L A p R A R L and A R R in a figure. This shows alpha p alpha R , 

and the lateral field angle range for the field angle range of the sliding direction which each camera 
21 covers by alpha p , alpha p R , and aipha R L gnd alpha R R . The straight line which connects the 

camera 21 to the above-mentioned driver, the slide position of a sheet and inclination of a reclining 
change with a driver's physiques and favorite postures - further -- each - direction being changed 
in connection with them and direction of the vertical plane containing the above-mentioned straight 
line also being strictly, changed, since it changes also with change of an occasional posture, but. 
The direction of the above-mentioned vertical plane, and the same direction as a parenchyma top, 
What is necessary is just the direction in the range which permits a gap of such a driver's position, 
and a gap of the direction accompanying change of a posture, it is the technique of deducing 
statistically the standard position which considered ail such elements, and determining a driver's 
position, and what is necessary is just to set up a direction according to it. 
[0041 ]And if it installed so that the horizontal axis on the image of the camera 21 might become 
level as it is, With reference to drawing 4, for example about camera C p L> bumper corner 90' of a 

self-vehicle will be located in the central lower part of a screen, and will be an image to which white 
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line W L of the road surface which is parallel with self-vehicles crosses a screen in the shape of a 

diagonal line in the direction of the upper right (or upper left) from the lower left (or lower right). 
Then, the camera 21 is installed in the posture which gave the inclination to the circumference of 
the optic axis X as the 3rd device of an installation attitude. For example, about a left-hand side 
camera, a right-handed-rotation inclination is attached by twisting on the right at the circumference 
of an optic axis to vehicles. Thereby, a screen serves as direction whose feeling of a driver a 
forward direction suits as shown in drawing 4 . Picture P p , is specifically a picture of camera C F L 

of installation to a forward left corner, Forward left corner 90' is copied in a lower right corner, linear 
shape white line W' L L to a forward direction is reflected in general to an inclined upper right 

direction in a straight line from a middle-of-the-screen lower part, and feelings come to suit an 
angle and depth perception when left-hand side white line W L of a self-vehicle is actually seen 

from a driver's seat. If it says, when picture P p of the front left and picture P p R of the last right 

have been simultaneously arranged to the right and left of a screen, this picture, for example Self- 
lane left-hand side white line W' L on picture P p at that time, Self-lane right-hand side white line 

W, „ on picture P,_ _ is the arrangement which seems to cross depending on the method of infinite 

L R F R 

distance like [ when the front is photoed with one camera ]. Thus, the inclination given to a screen 
is the same also about right-hand side and back right and left. 

[0042]The example of attachment of camera C pL to a forward left corner is shown in draw ing 5 and 

drawing 6, and (the installed position of an imaging device), next the installed position of an 
imaging device are explained in detail. In this example, camera C p has the headlight 91 and 

composition included in the really constituted side lamp 92, as shown in a figure, and it makes 
attachment cheaply possible by this structure at slight change also at the existing vehicles. It is 
also possible to make a lamp small with a natural thing, to include in the surplus space, to include 
in shells other than a lamp, or to attach to a car body surface directly. Although a vehicle 
circumference and the method of infinite distance are incorporated by this simultaneously [ with the 
self-body forward left corner outermost edge (a figure lefthand corner part of a front bumper) 90 of 
a camera C p , directly under ] as mentioned above, In order to incorporate the widest possible 

range, change also with types of a car, so that the above-mentioned terms and conditions may be 
satisfied, but. it is a little larger than about 90 degrees to it in general about field angle alpha p of a 

sliding direction, and camera C p of a wide-angle lens which hears from it a little from about 90 

degrees in general also about field angle alpha p of a longitudinal direction in size, and can secure 

a grade is needed. However, since the gap with a sense of distance of the substance which a 
driver looks at will become large if a field angle is made large not much, there is a limit naturally. 
Therefore, in this embodiment, camera C._ , which made 125 degrees 97 degrees and field angle 

r L 

alpha p L of the longitudinal direction for field angle alpha p of the sliding direction is used as an 
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example. 

[0043]an installation attitude and an installed position natural picture profit ****. [ thus ] The 
detailed example of picture P p of camera C p L of a self-vehicle front left corner is shown in 

drawing 7. How where the picture at the time of doubling the forward left corner of a self-vehicle 
with the vertical upper part of the intersection of white tine W L of the direction of movement drawn 

on the road surface and white line W L which intersects perpendicularly with it by a diagram is in 

sight is expressed. In this screen, on the basis of bumper corner 90' of a self-vehicle, right-hand 
side white line W 'self-vehicle slow lane L including . ' of the direction of movement of a self- 

LK 1 

vehicle, it is possible to overlook the wide range to the lane L 'white line W which and also 

intersects perpendicularly' . n left lateral on the left, and to copy the method of infinite distance (a 

dashed line shows to a figure) to the whole right and left width of a screen. And the method of 
infinite distance on extension of the transverse plane of vehicles, i.e., the cross-direction medial 
axis of vehicles, is contained in the method of the infinite distance reflected to the whole right and 
left width in this way, and the method of infinite distance of this transverse plane is on a monitor, 
and it is horizontally displayed above bumper corner 90' of self-vehicles. 

[0044]The (image display method of a control device), next the monitor as a displaying means use 
the display 5 of the navigation device in this gestait. Such a monitor is installed near [ installment 
panel ] a driver's transverse plane. And feeling in case a driver looks in at this screen will be 
naturally recognized to be the front of a direction of movement as all of a traffic sign, a map, etc. on 
a road are considered as the notation the upper part of a display surface indicates the front to be, 
at the same time it realizes the screen upper part to be a physical top. Being recognized as back is 
also natural at the same time a screen lower part realizes as the physical bottom. Then, a screen 
display of this invention is made on the principle of such recognition. 

[0045]A actual method sets up the posture of the camera 21 so that each picture shown in drawing 
7 may be acquired. Specifically, direction of camera C R of vehicles back and C R R is attached to 

what is called upside down. Namely, as are shown in drawing 1 , and a driver sets an optic axis by 
the perpendicular line in the state where it sat on the driver's seat, first and the body front turns into 
the screen upper part, camera C Q which looks down at right under is supposed, Camera C F L 

leaned the optic axis to the front and a left from the state, and also leaned to the circumference of 
an optic axis at the right, and camera C p leaned the optic axis to the front and the right direction 

similarly, and also leaned to the circumference of an optic axis at the left. Similarly, camera C R L 

X nn leaned the optic axis to back and right and left, and also twisted to the circumference of 
and R R r 

an optic axis at right and left. However, inclination of a cross direction and inclination to a 
longitudinal direction are shown in a figure, and torsion of the circumference of an optic axis is not 
shown. If it is set as such a posture, a desired screen will be obtained only by displaying on the 
display 5 as it is. Although considering it as a mirror image which is looked into with what is called 
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a reflector glass of photoing back with a camera, carrying out flip horizontal processing of that 
picture information about this point, and displaying on a monitor is also considered, although a 
system is complicated, since it does not suit sensuously, such a gestalt is not taken with this 
device, in this way, the picture acquired comes to agree in direction of actual white line W L L which 

white line W, . , W . „ of the right and left on a screen looked at from the driver's position, and W 

L L L R 

D , as shown in drawing 4. 

[0046] In this way, the picture information obtained shall be chosen according to a place, a 
situation, etc. by making to stop the display to the necessary minimum according to a driver's 
demand into essential principles, and the driver shall be provided with it. The process flow of the 
whole system constituted in accordance with this meaning is shown in drawing 8. ********** shown 
with the numerals A which this system was fundamentally divided in accordance with the kind of 
driver operation, and attached O seal on the figure, it comprises an intention which supports the 
obstacle avoidance operation similarly shown by B, the parking operation shown by C like the 
following, the blind corner operation shown by D, the back dead angle check of E, and the white 
line check of F, respectively. 

[0047]At the first step S-1 , data read in is performed from the input device 2 which constitutes the 
system of drawing 2 that the assist content of above-mentioned A-F should be realized. And the 
function which operates in consideration of safety and necessity by a speed region at the following 
step S-2 is divided. Namely, when judgment of a low vehicle speed region is materialized from the 
input of the speed sensor 25, When an one judgment of the selection SW (switch) is made and this 
is materialized at the following step S-3, a screen display according to selection of ********** of A, 
obstacle avoidance operation of B, the parking operation of C, or blind corner operation of D is 
performed. On the other hand, when judgment of Step S-2 of a low vehicle speed region is an 
abortive junior-and-senior-high-schools vehicle speed region, display processing of the back dead 
angle check of E and a white line check of F is performed. What is necessary is just to set up those 
concrete setting out individually by test evaluation etc., although a different standard for every 
function is required about a detailed speed region. 

[0048]Next, when switch selection [ which / of A-D ] is made in the low vehicle speed, the function 
corresponding to the selection SW operates. Since these functions are functions of another 
purpose which became independent fundamentally, multiple selection is constituted so that it 
cannot do. Even when the switch is not chosen, the system is constituted so that it may operate 
automatically in consideration of safety about the display function of obstacle avoidance operation 
of B, and blind corner operation of D. Numerals B' which attached 0 seal in the figure, and D" show 
these, in the case of a junior-and-senior-high-schools speed range, from the function of a white 
line check of F, it leads to the function of a back dead angle check of E, and goes. These individual 
functions are explained below. 

[0049](**********) The method of presentation in **********, As shown in drawing 9, outermost line 
W n . of vehicles and W Q _ are taken down at right angles to a ground surface, and it is based on 

BL B R 
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displaying in piles line (parallel to vehicles media! axis) l_ B L which extended the line to the vehicles 

cross direction, and L D _ on a screen. In this case, the outside line of a tire may be displayed 

instead of outermost line L„ , and (_„ n . These outside extension line can aiso be considered as 

the line which did not necessarily need to double with the outermost part of strict vehicles, or the 
outermost part of the tire, and gave the margin of the about [ 20 cm ] from those outermost parts. 
As an option, as shown in drawing 10, it corresponds to the amount of steering rudder angles, and 
how to display forecasted-locus L Q of the outermost line of vehicles or the outside line of a tire 

and L_ c as shown in (1 ) of a figure or (2) is also considered. How to display this forecasted locus 

is explained in detail in the place of obstacle avoidance operation. About the pattern of a display 
line, various gestalten, such as considering it as two or more lines about a single straight line or a 
10-cm interval, as shown in d rawing 9 , or attaching the picture of a tire, making a tire outside line 
imagine, or making it a three-dimensional expression so that a ground surface and the lateral 
surface of vehicles can be imagined, can be considered. 

[0050] Next, the conditions which display carry out on condition of [ that a driver makes ******** 
selection ] the selection SW shown in drawing 2. The process flow by this selection is shown in 
dra wing 1 1 . ********** considers it only as the low vehicle speed, and it is not made to operate in 
the high vehicle speed. Such ********** j S dangerous at the time of high speed operation, and it is 
because the picture information itself may distribute a driver's attentiveness. First, when meaning 
bringing near by the slot in left-hand side as much as possible, and stopping during an advancing 
travel, a driver chooses the flush left SW. This selection is judged by step SA-1 . Since under an 
advancing travel will be materialized by judgment of step SA-2 if it becomes below the vehicle 
speed beforehand set up in the state, processing picture P p of step SA-4 is displayed. Then, a 

driver is carrying out operation so that leftmost outside line l_ B of the self-car to superimpose may 

be doubled for the purpose of the slot on a screen, and he can ******** easily, if ******** finishes 
and a shift iever is put into the "P" range position, the input of position SW{switch)23 by it will be 
judged by step SA-1 1 , the selection SW will be canceled by step SA-1 2, and ********** will be 
completed. Otherwise, as for this condition resolutive, beyond as for vehicle interdiction fixed time, 
a shift lever is considered [ beyond vehicle interdiction fixed time or engine shutdown j by brake 
SW (switch) one in the "N" range position. If a shift iever is made into the "R" range position when 
it is necessary to go astern by a situation in the middle of **********, by judgment of step SA-2 by 
the input of position SW(switch)23, a picture will replace P R by step SA-5 and it will become 

sternway aiding pictures. Especially in sternway, since the front left corner of vehicles sways right 
and left, it is effective to take the method of presentation which displays picture P p and P R L 

simultaneously as shown in the upper right of drawing 11 . Although the above explanation is 
premised on the general front-wheel steering mechanism, in the case of the four-flower steering 
mechanism adopted as some cars, it is effective also at the time of advance to consider it as 2 
picture simultaneous display, as shown in the drawing 1 1 upper right. 
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[0051]On the other hand, flush right judgment of step SA-1 is materialized by selection of the flush 
right SW and it becomes a screen display of step SA-6 or step SA-7 according to pre-sternway 
judgment of step SA-3 to bring near vehicles by right-hand side by a situation. When passing 
through the place which is narrow with the time of passing on a narrow way, the obstacle, etc., If 
********** sw is chosen, by judgment by step SA-1 and step SA-8, the right-and-left picture of one 
of front and sternway can be simultaneously displayed on one screen by step SA-9 or step SA-10, 
and support suitable for operation can be offered. Omitting a part of incorporation picture except 
left brink ******** in drawing 11 , it is what simplified and showed the screen and a actual display 
screen contains an image pick like left brink ********. 

[0052](Obstacle avoidance operation) The method of presentation in obstacle avoidance operation 
displays passage forecasted-locus L g of the forward left corner according to a steering rudder 

angle, as shown in drawing 12 (example which avoids the car under parking in front by the picture 
of a forward left corner). With a natural thing, it distinguishes by forecasted-locus I_ B Q f at the time 

of advance, and forecasted-locus l_ B R at the time of sternway, and displays. Forecasted-locus 

L in this case displays a line not the assumption line on a ground surface like [ in ********** ] but 

B S 

on the basis of a lefthand corner part outermost part. As shown in drawin g 12, it is the purpose to 
avoid obstacle N', and this reason is for making it intelligible for a driver. In the case of this screen, 
as shown in drawing 13, how to display distance line l_ K used as the rule of thumb of an interval is 

also considered. When dark in the outside of a car and the color of the self-body is a dark color 
system, when the shadow of the self-body or other objects is reflected in a road surface, it may 
become difficult conversely with the light of the sun or a lighting to distinguish the outline of the 
self-body. It will become very intelligible, if it puts on the outermost edge of self-body 90' and 
border-line (outermost bridle wire of self-vehicle) L p of the self-body is superimposed on a screen 

a sake [ in such a case ], as shown in drawing 12. As for this, the same may be said of the case of 
other functions. 

[0053]The conditions which perform the display in this case are limited as well as the case of 
********** according to the process flow shown in drawing 14 at the time of the low vehicle speed. 
When a driver operates selection SW(s) (for example, front left etc.) shown in drawing 2, it is 
judgment of step SB-1 according to it, and the picture (for example, drawing 15 ) of a corner 
corresponding by step SB-2 is displayed. Based on the input of the steering rudder angle sensor 
26, forecasted-locus l_ B p of a steering rudder angle and the body corner corresponding to order 

** and L D e _ are then displayed collectively. This display is terminated by release processing of all 

B S K 

the corners SW by step SB-4, when all the detection distance of the distance sensors 28 becomes 
beyond a reference value by judgment of step SB-3. When a switch is not chosen, it progresses to 
B" by judgment of step SB-1 , The detection distance of the distance sensors 28 of each corner 
becomes below or more one reference value by judgment of step SB-5, and when the distance is 
approaching further by judgment of step SB-6, while displaying the picture of an applicable corner 
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by step SB-7, it warns a driver by step SB-8. An example of the display screen in that case is 
shown in drawing 15 . By a diagram, a front left corner is taken for an example, the front left corner 
of picture M { of a self-vehicle is displayed with precautions, or picture P p L of a front left corner is 

displayed on the position which makes it blink and corresponds. Warning may be further performed 
to a sound and a sound. Since there may be plurality, the corner applicable with a natural thing 
should just add two or more pictures and warning all over the screen of drawing 15 in that case. 
[0054] {Parking operation) Here explains as an example the case of retreat parking said to be the 
most difficult by parking operation with reference to drawing 16 . In this operation, a target parking 
space will be checked in the position of **, and it will move forward to the parking operation starting 
position of **, and will stop to the parking position of ** through the position of ** by sternway. The 
points which park a car with sufficient performance at this time are how the parking operation 
starting position of ** is judged, and at what to aim in process of **. It is a position of ** in principle, 
and when target parking-space U has entered inside (it is the upper part on a figure) from the 
minimum turning radius R of the vehicles, a car cannot be parked by once but cutback operation is 
needed. As a general rule of thumb, since the minimum turning radius of the usual passenger car 
is around 5 m, the method aiming at the car L-2 {position assumed when there was nothing) of a 
two-set next door of a parking space is used. Most drivers depended on a habituation and 
admiration, and have decided the position of **, and it is difficult to dedicate without cutback 
operation to the position of **. 

[0055]To then, the display on the monitor of the picture from which the method of presentation in 
this case is incorporated with the camera 21 . either of by the side of the right and left of the parking 
required space containing a part for the margin space needed for vehicles right and left when it 
reaches by sternway which makes a handle the maximum rudder angle ~ a near side {seeing from 
vehicles in retreat parking to the left rear left-hand side.) In retreat parking to the method of the 
right rear, the range which can be parked bordering on right-hand side is superimposed, seeing 
from vehicles. If the case of retreat parking of this to the left rear is explained concretely, as shown 
in drawing 17 , by parking operation starting position **, parking-space range Z, will be displayed on 

an after [ the left ] picture. When a steering is cut from parking operation starting position ** to the 
maximum and is dedicated to target parking-space U of drawing 1 6 as mentioned above at the 
posture {it is a right-angled posture to **) of **, this parking-space range Z y It is what is set up as a 

possible field which can give a margin to vehicles right and left, The left-hand side limit in which the 
boundary line of the transverse direction shown in d rawing 17 had a predetermined margin to the 
vehicle side face {the concept of this limit) When it went back and expresses temporarily to the 
position at the time of position empty vehicle both retreat starts of a actual display, a fictitious 
outline comes to show to a figure. The boundary line of a lengthwise direction shall show the limit 
that it becomes a posture of ** and a predetermined margin can be taken to a vehicle front. It sets 
up so that display shape may suit on the basis of white line W' p of target parking-space U' {set up 

leave a predetermined gap to front and rear, right and left of vehicles in this very thing and a 
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parking position) specifically reflected in a screen as shown in drawing 18 . A setting method shall 
be based on an experiment or shall be based on calculation from the design basis of vehicles 
specifications, the specifications of a camera, the attachment specifications to the vehicles of a 
camera, display specifications, or a motor pool, etc. If target parking-space U' goes into parking- 
space range Z with a margin in actual operation as shown in drawing 17 w hen it regards as a 

screen by parking operation starting position **, receiving in all directions [ of white line W' p ], as 

operation will also have a margin, and will be possible and it is shown in drawing 18 - the 
boundary of parking-space range Z ( in every direction - about - if I am doing one, it will be 

dedicated by the sternway which cuts a steering to the maximum by the posture which does not 
have a position and inclination barely. Therefore, as for a gap in the lower part of parking-space 
range Z. from the position shown in drawing 18 , parking operation starting position ** expresses 

back past ******, and a gap to the left means that a parking operation starting position visits left- 
hand side too much. 

[0056]Next, when resulting in parking position ** through the position of ** from parking operation 
starting position ** on the way, as shown in drawing 19, the picture of back right and left is 
simultaneously copied on one screen. As shown in a figure, picture P R L of the left rear is arranged 

in left-hand side, picture P D D of the method of the right rear is arranged in right-hand side, and 

K K 

outermost line l_ B and L g are displayed on each like screen P f R of .dra^ngJ3, and P R R . It may 
be made for this line to display forecasted-locus L D _ _ according to a steering rudder angle. What 

D O K 

is necessary is to set up so that feelings may seethe with a driver, but to carry out test evaluation 
and just to decide the display interval of a picture on either side. Although picture P R Land P R R are 

respectively independent pictures, since they have the wide-angle lens that each looks down at a 
self-body corner and the method of infinite distance, the arrangement of the picture shown in 
drawing 19 can see in piles the range shown by R of drawi ng 1. Therefore, the gap of two pictures 
can be taken or very easy processing of cutting and putting the inside in order can constitute back 
false image composing. 

[0057]lt becomes intelligible by displaying target parking base line W , as shown in drawing 19. 

This base line W ( superimposes the parking-space shape at the time of parking vehicles at a 

standard parking space finely on a screen. Namely, if it is operated so that target parking base line 
Wj may be doubled during parking operation at target parking-space U as shown in drawing 19, At 

the time of parking operation completion, to target parking-space U of the self-body, a gap on 
either side, the position of order, and inclination can grasp clearly, and parking also of an unripe 
driver is attained finely. A driver does operation so that this base line V\l { may be doubled with 

target parking-space U' reflected on the screen, these displays - in addition, if an alarm is emitted 
and it is made to carry out image display with the obstacle sensor 28 and the camera 21 of front 
and rear, right and left of vehicles which are shown in drawing 3 like the case of obstacle 
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avoidance operation of B when it is likely to contact, safety can be improved more. Although this 
parking operation supporting technology is applicable similarly [ in parallel parking and inquiry 
parking ], it omits explanation here. 

[0058]The conditions which perform the display in this case follow the process flow of the operation 
shown in drawing 20. Also in this case, it shall support like said two operations only at the time of 
the low vehicle speed. This flow is started because a driver chooses the parking SW shown in 
dr^ingjk the case of left rear parking shown in drawing 17 -- a driver - SWP L - in right rear 

parking, SWP R will be chosen. When judgment of step SC-1 according to this selection explains in 

the case of left rear parking, judgment of step SC-2 in the input of the position switch 23 at the time 
of advance. Parking-frames range Z ( explained to left rear picture P RL by drawing 19 by step SC-4 

is superimposed, and if operation is carried out so that target parking-space U" with a driver actual 
to this range Z f may enter, it can go to the position of ** explained by drawing 17 w ith vehicles. If a 

driver checks this state and puts a shift lever into sternway, a screen display of back two picture P R 

L and the P R R will be simultaneously carried out by step SC-5, and forecasted-locus line L g g R 

according to a steering rudder angle will be superimposed to each corner. A driver refers to this, 
and he does operation so that it may go into actual target parking-space U' reflected in a screen. 
This flow is terminated by switch release of step SC-9 by the shift judgment to the parking range of 
step SC-8. 

[0059](Blind corner) The picture selection which relates to the theme of this invention and displays 
the incorporation picture of the information on the crossing road on a monitor automatically is 
applied to the blind corner display. As shown in drawing 21 (example of picture P f in the case of 

the direction blind of the forward left), the method of presentation in the case of a blind corner 
copies the picture of a blind part, and displays the rule-of-thumb depth mark on the ground surface 
from a self-vehicle as milepost l_ K on it. The last right and back right and left are also the same. In 

this case, as shown in Screen (1 ) or (2) of drawi ng 22 , according to order **, a picture on either 
side may be displayed simultaneously. And the symmetry of the right and left in the case of the 
simultaneous display of a right-and-left picture is taken into consideration in this way, Add some 
change about the posture of the camera explained at the beginning, and about the optic axis X of 
the camera 21 , As opposed to the camera 21 (for example, C p ) of right-and-left one corner of the 

vehicles installed in the posture in which it turns to the vertical plane direction containing the 
straight line which connects the camera 21 to a driver, and the same direction as a parenchyma 
top, Kazunori also installs the camera 21 (for example, C_ R ) of the corner of right-and-left another 

side in the posture of plane symmetry to the medial axis of vehicles, if picture M ( of a self-vehicle is 

superimposed so that it may correspond to the middle of the screen at that time forward and 
backward, it will become much more intelligible. 

[0060]The conditions which perform the display in this case follow the outline process flow of the 
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blind corner display shown in drawin g 22 . As said main flow explained this flow, it shall operate 
only at the time of the low vehicle speed. And on condition that the blind SW (the front left, the last 
right, back left, back right) shown in drawing 2 is chosen by judgment of step SD-1 , it carries out, 
According to front and sternway by input judgment of the position switch 23 of step SD-2, the 
singte-sided screen shown in drawing 21 by step SD-3 or step SD-3 which constitutes a picture 
selection means and an index displaying means, or Screen (1) or (2) of a figure is displayed. If it 
explains in the case of a screen (1), front left picture P p and front right image P p R will be 

arranged in picture M i of a seif-vehicie, and the position to which it is made to correspond, and 

milepost l_ K generated by the index displaying means will be superimposed. And this milepost l_ K 

also shows the driver's gaze direction, and the point at the time of a gaze is intelligible, in a figure, 
numerals N' shows a field-of-view obstacle. 

[0061]By the way, there are many crossings, also when the direction P of the road which crosses 
to the direction O of a self-vehicle becomes slanting like the example of drawing 23, and it not only 
intersects perpendicularly, but in such a case. It will become more intelligible, if milepost l_ K 

according to a crossed axes angle is displayed on each picture on either side as shown in dra wing 
24. display direction beta L Qf mj|epos{ . jn cgse the int ersecting angles of the example of a figure are beta - 

„ , ■ a * - * k beta n . This value is 

contrast with dashecHine which shows the milepost in an orthogonal crossing point shows R 

K 

uniquely determined as beta understood from the data of navigation from the specifications of the 
camera 21 , the loading specifications to the body, etc. Although the explanation in this case makes 
the straight iine the crossing road, and it may have bent or may have broken, according to that 
situation, the display direction and display distance of milepost i_ K are set up. In this case, if the 

direction sensors 29 {refer to drawing 2) carried in the car, such as a gyroscope and compass, 
detect direction of a self-vehicle with sufficient accuracy, the direction of milepost L K can be 

displayed with sufficient accuracy combining the map data of the database 4 (refer to drawi ng 2). 
The driver can refer to this picture, and can check safety on either side, and it can move forward. 
[0062]Even when the blind SW is not chosen by judgment of step SD-1 according to the feature of 
this invention, perform processing of D' and by step SD-7. When it judges going into the crossing 
which does not have a signal by the Navi information based on the database 4, image display is 
carried out similarly and operation is supported. In this case, in addition to the traffic information 
from a navigation device, approach in the crossing which does not have said signal based on the 
traveling condition of the vehicles inputted from the input device 2 and the information on the 
contents of operation can also be judged. Make the information on the traveling condition of the 
vehicles at this time into the low vehicle speed information by the speed sensor 25 included by the 
input device 2, and let the information on the contents of operation be the brakes operation 
information by one of brake SW24 similarly included by the input device 2. Even when step SD-7 is 
abortive, by step SD-8, also when the detection distance of the obstacle sensor 28 is below a 
reference value, it judges that a prospect is bad, and a picture is displayed. 
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[0063](White line check) Generally, although a driver runs checking the self-vehicle position on a 
travel lane by that cause looking at the white line on a road surface, when running daytime also at 
the time of fog, etc. night and while it is raining further or, his lane may be unclear and he may take 
pains over the localization. The white line which displays the center line serves as an important 
verifying means which judges not only the localization but the propriety of passing each other at 
the time of passing each other by the road of single-sided 1 narrow lane of the width of road. 
Therefore, it is also dramatically dangerous if a white line cannot be recognized. Support of a 
check of such a meaning to a white line is important. 

[0064]ln this gestalt, a white line check is supported that such a situation should be coped with. 
And it shows it. [ drawing 25 ] [ the equipment configuration of the image pick-up part in the case of 
copying white line W L , on the left-hand side of vehicles ][**][ type ] In this case, as an imaging 

device, camera C p of the corner of the vehicles front left part shown in drawing 1 is used. If lamp 

92* which irradiates the corner of a vehicles front left part with the direction of the left and the 
ground surface of the method of the forward left is similarly provided (this lamp is used when 
vehicles have been equipped with side lamp 92') and white line W L L is illuminated, white line W" L L 

can be recognized visually more vividly. Visual recognition of white line W L is attained without 

being dazzled by the light of the headlight of an oncoming car rf it does in this way. 
[0065]The process flow of a white line check is shown in drawing 27. As conditions, it is more than 
a certain fixed vehicle speed by Step S-2 of the main flow shown in drawing 8, The headlight 
lighting judgment by step SF-1 by this flow is materialized, The time of the wiper operation 
judgment by step SF-2 being realized in one of wiper SW24 Night, The inside of rain is judged to 
be under a run, and at the time of blinker SW the non-operating judgment by step SF-3, a side 
lamp is turned on by step SF-4, it irradiates with a front road surface from a left lateral, and the 
screen of drawing 26 is displayed by step SF-5 which constitutes a picture selection means. 
Although not shown in a process flow, by judging low visibility by lighting judgment of a fog lamp 
being materialized, or strong operation judgment of a wiper being materialized as other conditions, 
above the same fixed vehicle speed, the low visibility by a heavy rain is judged and same 
processing can be performed. In the above-mentioned flow, when the judgment in rain or judgment 
of low visibility is abortive, at night, It moves to process flow E of the back dead angle check which 
carries out a postscript, and even when both the above-mentioned judgment is formation, the 
requirements for judgment of the flow of a back dead angle check will be ****(ed) at the time of a 
blinker operation, and it will display back dead angle screen E L or E R . Side lamp lighting is not 

necessarily a necessary condition. Since the optical axis direction of camera C p L of {he front |eft rjght 

C p is large and they differ to the direction of headlights, such as an oncoming car, Even if the 

light of an oncoming car carries out specular reflexion to the water screen, it does not go into CCD 
of a camera, but it is not dazzled and is because the catoptric light from a white line can be caught 
by CCD since it is scattered reflection. Therefore, it is fully a little detectable from the side of a self- 
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vehicle to the front using the lighting of an oncoming car, a following car, the next car, a self- 
vehicle, etc. Thus, side lamp lighting is for making it further easy to be visible. 
[0066]As a display screen, as shown in drawing 26 , the picture of camera C p of a front left corner 

and camera C.. _ of a front right corner part is copied on the same display. It is desirable to set up 

F R 

become the same as the time of taking a photograph with the camera whose relative relation of 
white line W\ . , W , D on either side is one so that feelings may seethe with a driver at this time. If 

L L L R 

border-line L p F of the self-vehicle is superimposed in the similar way with previous obstacle 

avoidance having described since the self-body within a picture can become difficult to be seen 
when the color of the self-body is dark, distinction with a road surface will become easy. 
[0067]by the way, under such an unfavorable condition, a camera image is displayed as it is - a 
picture tends to become indistinct if it is accepted to the driver under a next door and operation, 
it may be hard to recognize. Then, it is also effective to make it be easy to recognize by the 
following methods. 

[0068]The modification shown in drawing 28 is the method of superimposing the figure which 
carries out highlighting which makes a white line conspicuous by image processing, or expresses a 
white line on a picture white line. In this case, after display processing of step SF-5, white line 
recognition and inclination will be recognized by step SF-6, and processing which superimposes 
the figure which displays emphasis or a white line for the white line on a picture by step SF-7 
based on it will be performed. On a drawing, it is expressed as the dashed line. A foreground color 
does not adhere to the white doubled with the white line, but should just use the color which is 
easy to recognize visually to a driver. 

[0069]Next, drawing 29 shows the 2nd modification that displays a white line figure with the HUD 
(HUD) which used the windshield. Also in this case, after display processing of step SF-5, white 
line recognition and inclination will be recognized by step SF-6, and HUD display processing will 
be performed by step SF-7' based on it. Let a display position be a position which laps with the 
white line position doubled with the driver's actual look. Since the driver can check a white line 
position like real according to this method of presentation, without diverting a look, with driving 
view maintained, the advantage whose safety improves is acquired. 

[0070]When the white line which was advanced further and detected separates from a display 
function from white line tolerance level, or when it is separating, it warns a driver of the 3rd 
modification shown in following drawing 30. The case where a self-vehicle is separating on left- 
hand side by a diagram is illustrated. As processing in this case, after display processing of step 
SF-5, recognize the white line recognition and inclination by the step SF-6 [ same ] as front 2 
example, and That result, It is comparison with the white line tolerance level beforehand set as the 
fixed position on a screen to the self-vehicle position, and by judgment of step SF-8, from tolerance 
level, the white line of an image should just warn a driver by step SF-9, a blank or when it becomes 
that it is likely to separate. Although various concrete warning methods are assumed, there are the 
methods of carrying out a character representation, like "it has separated on the left", for example 
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like a graphic display. 

[0071]By the way, the method of utilizing white line detection art for rain blank warning etc. is 
already proposed [ various ]. as for the conventional camera for white line detection in such art, it is 
common to be considered as the arrangement which photos a front white line from the 
neighborhood (usually reflector glass neighborhood) of a driver's seat over a bonnet to dra wing 31 
and drawin g 32 as cable address CCD 0 shows. When it does in this way, an image pick-up can be 

made to approximate to the viewing angle at the time of a driver's operation mostly, and there is a 
big advantage in that a white line on either side can be seen simultaneously. However, in this 
method, there is a problem with the point that see a white tine to a shallow angle and a side face 
direction to a road surface, and see to angle theta Q v and a plane direction, and are angle theta QH 

and a distant place will be seen. That is, if it will be in the state where a water curtain is made into 
a road surface in a rainfall, as [ show / in a figure ], seeing at a shallow angle will become an 
obstacle and it will become difficult to see the white line of the road surface under it. If especially 
confused by the catoptric light of the lighting of the rear light of the headlight of an oncoming car, or 
a preceded vehicle, a street light, etc., it will become impossible also on the image pick-up with a 
camera to white line detect a driver the same with the ability of a white line not to be seen on a real 
field of view. It becomes an obstacle to see a long distance on the contrary, in the case of fog, 
smoke, etc., it will see through the layer of a thick fog or smoke, and it becomes too impossible to 
white line detect them of the road surface in a camera. When a driver is not seen rain blank 
warning, just by being made appropriately, it is worthy, but if detecting becomes impossible 
together with a driver by the above camera arrangement, the value as a supporting system is not 
useful. 

[0072]About this point, according to the white line check of the above-mentioned embodiment, a 
visual recognition improper situation can be detected, and the road surface state of self-vehicle 
nearness which is not confused by such state can be detected and supported. In this case, camera 
CL 0 (cable address CCD U shows at dr awing 31 and drawing 32) of the forward right angle set up 

PR H 

to have stated to the beginning based on the basic thought of this invention has a meaning 
important for the point moreover looked down on at a vertically near angle to a road surface from 
the position near a road surface. Angle theta H which looks at a ground surface in a side face 

direction as shown in drawing 31 w ill be very large (theta R v »theta Q ) almost vertically near, 

therefore will look at the road surface very near the self-body. As a plane direction is also shown in 
drawing 32 , a possibility that angle theta H H will be confused by the lighting of an oncoming car 

more greatly (theta u >>theta^ J enough than angle theta^ u becomes low. It sets up in this way, 

and for the first time, also in the case of a frightful heavy rain at night and a thick fog T a white line 
can be phoioed, a driver is provided with a picture, and it becomes possible to recognize the 
position of a seif-vehicle to a lane. If image processing is carried out like the modification 1 or 2, 
white line recognition can be made still easier also under such an unfavorable condition. An exact 
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warning like the modification 3 is also attained. Although illustrated about camera C pR (CCD H ) of a 

forward right angle by a diagram, it can say that the same may completely be said of the case of 
the camera of the forward left angle which omits a graphic display. 

[0073] Drawing 33 shows theoretically the relation of how where the subject of the sea bed by the 
catoptric light and the underwater transmitted light in the water surface appears. Correlation is 
realized between the intensity of lights which carry out surface reflection like a graphic display with 
the angle theta which looks down on the water surface, and, as for the underwater transmitted light 
shown in a figure as a solid line, strength is reversed bordering on angle theta 1 to the catoptric light 

in the water surface shown in a figure with a dashed line. Therefore, in an angular area smaller 
than angle theta^ it will be in the state where a sea bed can be seen without being interfered by 

water surface catoptric light, to being interfered by water surface catoptric light and a sea bed 
being unable to be seen in a larger angular area than angle theta r Taking in of the image in an 

angular area smaller than angle theta 1 in this case is in charge of the image pick-up by the above- 
mentioned conventional camera CCD 0> and taking in of the image by camera CCD H in a larger 
angular area than angle theta is in charge of the image pick-up by the above-mentioned 

embodiment. According to the above-mentioned embodiment, the white line check under bad 
weather is enabled using such a principle. 

[0074]{Detection of a white line) In addition, the operation supporting device of this embodiment is 
applicable also to detection of a white line. About the concrete technique of white line detection, 
since it is common knowledge, explanation is omitted, but those many are installing the camera 
near [ windshield ] the car interior of a room for exclusive use as mentioned above. White line 
detection can be made to perform with this device to it using two cameras of the vehicles front left 
right corner part provided in dead angle detection as mentioned above. An advantage, such as 
being able to detect at night also at the time of a rainfall and fog, and being by detection of the 
white line beside [ using the size of the field of view of the transverse direction which is the feature 
of a run proper support device by carrying out like this ] the self-vehicle neighborhood, as the 
display of the aforementioned white line explained while relative position relation with the self-body 
is detectable with sufficient accuracy, is acquired. And it becomes cheap for other functions and 
combination. 

[0075](Back dead angle check) The theme of this invention is started and the picture selection 
which displays the incorporation picture of the information on vehicles back on a monitor 
automatically is applied to the back dead angle check. The flow of a display of the back dead angle 
check connected with previous white line detection is shown in drawin g 34 . This display by the 
judgment of step SE-1 which is more than a set vehicle speed and the Navi information shows Two 
or more lanes of one side, when the blinker operation by step SE-3 is judged during a run in the 
place out of which it does not come near the crossing by judgment of step SE-2, When it performs 
by judging it as passing or interruption and blinker operation is not judged, screen selection by 
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processing of E 1 is performed according to the action of vehicles. When there is blinker operation 

of the left or the right, a back picture applicable by processing of E L or E R is displayed by step SE- 

4 or step SE-5. When there is no blinker operation, processing shown in d rawing 35 is performed. 
[0076] In processing of E shown in this drawi ng 35 , white line detection by camera C pL of the front 

left and the last right and C pR is performed by the original step SE-10, and when this detection is 

impossible, processing is ended, without performing a screen display in the course of E 2 as it is. 

Control-line R 1 of every two right and left on image recognition when white line detection is 

possible step SE-10, as shown in drawing 26 , R 2 , and l_ 2 are set up, and it judges by the 

motion of a self-vehicle image to these control lines. Control-line R 2 and L 2 are self-vehicles 

outermost extension wire here, and, as for control-line R 1 and L y prescribed distance is the left 

parallel lines from it. When it judges with the right white line having crossed R 1 on the left from the 

right by the following step SE-1 1 to the control line set up in this way, The speed crossed by step 
SE-1 3 judges that it is beyond a reference value, it progresses to E R noting that there is a lane 

change intention, when this is satisfied, and a described image is displayed. When judgment by 
step SE-1 3 is not satisfied and control-line R 2 is crossed by step SE-1 5, it already progresses to 

E D as under a lane change, and a described image is displayed. On the other hand, when the left 

white line crosses L 1 from the left by step SE-1 1 step SE-1 2 at the right in the case of no and the 

speed further crossed by step SE-1 4 is beyond a reference value, it progresses to E L and a 

described image is displayed. When the speed crossed also in this case is less than a reference 
value, it divides a display and un-displaying by the position judgment by step SE-16. Thus, the 
erroneous decision by wandering etc. of the vehicles which are not meant is avoidable by judging a 
driver's intention at a position and move crossing speed on the basis of the control line of every 
two right and left 

[0077]A driver refers to this picture and does operation. About lane judgment, lanes which are 
running the lane number of the road it is running, and the self-vehicle are easily known using the 
white line detection art described previously. Even if it uses the signal of the light beacon currently 
installed in the major thoroughfare, the same thing becomes possible easily. Using these 
information can also perform judgment of passing or interruption easily combining blinker 
operation. Of course, if it combines with the Navi information, naturally information precision 
increases. 

[0078]As mentioned above, although some modification was given and explained based on one 
embodiment for the facilities of an understanding of the technical thought of this invention, this 
invention is not limited to the embodiment or modification of illustration, within the limits of a matter 
given in each claim of a claim, can be boiled variously, and can change and carry out concrete 
composition. 
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[Translation done.] 
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